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Qr^ (57) Abstract: A method of computer aided treatment planning is performed by generating and manipulating a three dimensional 

(3D) image of a region which includes at least one anatomical structure for which treatment, such as surgery, biopsy, tissue compo- 
j nent analysis, pro thesis implantation, radiation, chemotherapy and the like, is contemplated. A virtual intervention, which simulates 

at least a portion of the contemplated treatment, is performed in the 3D image. The user can then determine the effect of the inter- 
^5 vention and interactively modify the intervention for improved treatment results. Preferably, a warning is automatically provided if 

the intervention posses a risk of detrimental effect. The user can navigate through the contemplated region in the 3D image and asses 

the results. The treatment plans can be saved for comparison and post treatment evaluation. 
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SYSTEM AND METHOD FOR 
COMPUTER AIDED TREATMENT PLANNING 

SPECIFICATION 



FIELD OF THE INVENTION 
5 The present invention relates generally to surgical planning and more 

particularly relates to a system and method of using three dimensional interactive 
computer visualization in surgical planning, optimization and evaluation. 

BACKGROUND OF THE INVENTION 

In many areas of medical treatment, it would be beneficial for a 

1 0 medical practitioner to be able to visualize a region for which treatment is 

contemplated and to accurately simulate the contemplated treatment. By visualizing 
the effect of the simulated treatment and altering the proposed treatment to optimize 
the results in a virtual setting, results can be improved and risks associated with the 
actual treatment can be reduced. This is particularly true in the case of invasive 

15 procedures such as surgery, biopsies and prosthesis implantation. The virtual setting 
would serve both as a tool for the guidance for actual treatment and as a "gold 
standard" for evaluation of the actual treatment and for follow up management. 

The ear is a specialized organ for which the computer aided treatment 
planning is expected to play a valuable role. The ear is an internal organ that is 

20 difficult to examine because it is encased in the temporal bone. The ear also contains 
important anatomic structures, including the hearing bones (ossicles), inner ear organs 
of hearing and balance, and facial nerve. Congenital aural atresia (CAA) is a 
congenital developmental anomaly of the middle ear that manifests with varying 
degrees of external auditory canal stenosis or atresia, ossicular derangements, poorly 

25 developed mastoid and tympanic cavities. This disease results in conductive hearing 
loss which can be severe. In some cases, however, CAA can be treated surgically. 
However, because of the complex anatomy of the ear and the risks associated with 
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considered an important element in surgical planning. Conventional two-dimensional 
(2D) CT images demonstrate the key anatomic structures of the ear, including the 
stapes, middle ear space, inner ear and facial nerve. However, the 2D CT is limited in 
its ability to represent the spatial relationships between these important structures. 
For example, an aberrant facial nerve, a retro-displaced temporamandibular joint, or a 
low-lying tegmen tympani might make surgical reconstruction difficult or impossible. 
Three-dimensional (3D) information would be very helpful for planning surgical 
therapy for treating congenital aural atresia and other forms of ear pathology. 

Traditionally, spatial anatomical relationships could only be surmised by 
mentally integrating sequential 2D CT scan images. However, the advent of 3D 
computer graphical visualization techniques and high-resolution CT scanning now 
allow 3D images (either surface-based or volume-based) to be constructed from 
sequential scan images and be displayed on a computer screen. Three-dimensional 
relationships between adjacent organs can be shown by interactively manipulating 
these virtual organs on the screen using a mouse or some other interactive devices. 
Over the past decade, many applications of such techniques in a number of areas of 
medicine, including otology, have been explored. 

The article "Three Dimensional CT Scan Reconstruction for the 
Assessment of Congenital Aural Atresia" by Andrews et al., which was published in 
the American Journal of Otology, Vol. 13, Num. 13, May 1992, discusses generating 
3D computer images from 2D CT data to aid a surgical planning of CAA. This paper 
demonstrates that the structures of the ear can be successfully rendered to assist the 
surgeon in visualizing the spatial relationships among the critical structures of the ear, 
such as the facial nerve, mandibular condyle, the locations and thickness of the atretic 
plate and the like. The system disclosed by Andrews et al., provided a useful 3D 
image of the anatomy of the ear. However, due to the complexities of the anatomy of 
the ear, additional computer-aided planning and 'Virtual surgery" features to confirm 
the surgical plan would also be desirable. 
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Accordingly, there remains a need for improved medical treatment planning 
tools for optimizing the procedures and for evaluating the results. 



SUMMARY OF THE INVENTION 
5 It is an object of the present invention to provide a system for 

computer aided treatment planning that employs 3D surface and volume rendering 
and a virtual intervention feature to confirm the proposed treatment plan. 

It is another object of the present invention to provide a system for 
computer aided treatment planning that provides warnings to a user if a proposed 

1 0 intervention presents risk of damage to surrounding structures. 

A method of computer aided treatment planning in accordance' with the 
invention includes generating a 3D image of a region which includes at least one 
anatomical structure for which treatment is contemplated. A virtual intervention is 
applied to the region of the 3D image to simulate at least a portion of the 

1 5 contemplated treatment The effect of the intervention, in terms of both efficacy and 
potential risk or collateral damage, can be assessed and the intervention can be 
interactively modified for improved treatment results. When the intervention plan is 
finalized, the user can navigate within the 3D image in the area where the intervention 
is applied to fully visualize the results of the intervention. The intervention plan can 

20 also be saved in computer readable storage for later use as a "gold standard" for 
evaluating the completed intervention at later dates. 

The present method is applicable generally to treatment planning, 
including surgical planning, virtual biopsy, and prosthesis implantation. 

In one application, the planning of surgery to correct aural atresia, the 

25 region in the image includes the ear and the at least one anatomical structure includes 
at least one of the temporal bone, facial nerve, and stapes. In this case, the virtual 
intervention includes the placement of a virtual cylinder representing the location 
where an external auditory canal may be formed. In determining the effect of the 
virtual intervention, the proximity of the virtual cylinder to anatomical structures of 

30 the ear is measured and a warning can be provided if the proximity is less than a 
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application includes changing the position and/or size of the virtual cylinder. 

Preferably, once the final position of the virtual cylinder is determined, 
a virtual drilling operation can be performed by removing the volume within the 
5 virtual cylinder and navigating through the region in the 3D image. The stages of 
placing an intervention, modifying the intervention, virtually performing the 
intervention and navigating through the results are both interactive and repeatable. 
Each plan can be stored in computer readable media as an entry in a database for 
future retrieval, examination and comparison. 

10 The manipulation of the 3D image can be costly and processor intense. 

Preferably, the step of generating the 3D image of a region includes acquiring a set of 
2D images of a region; applying segmentation to extract structures of interest; 
converting the 2D images to a voxel based dataset of region; storing the voxel based 
dataset in a partitioned data structure; and rendering the 3D image from the voxel 

15 based dataset. The partitioned data structure can take the form of a binary space 

partitioning (BSP) tree having a number of leaf nodes where the voxels of the voxel 
based dataset are stored. Using this form of data structure, the step of applying the 
intervention can preferably include: identifying those leaf nodes which are effected by 
the intervention; applying the intervention to the effected leaf nodes to launch voxel 

20 based constructive solid geometry (CSG) operation and image regeneration; and re- 
rendering only the portions of the 3D image associated with the effected leaf nodes. 
In this way, the aforementioned processing burden can be reduced. In addition, 
voxel-based constructive solid geometry (CSG) subtraction and level-of-detail 
rendering can also be used to reduce the processing burden. 

25 Preferably, re-rendering can take place based on a simplified level of 

detail (LOD) rendering mode. The CSG operation preferably includes converting the 
virtual cylinder into a voxelized format and performing a voxel-based CSG 
subtraction between the virtual cylinder and the tissue bounded by the virtual cylinder 
volume. 

30 The method of computer aided treatment planning can be used in 

connection with numerous contemplated treatments, such as biopsy. In this case, the 
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- intewention-is-the-placemento£a-^^ where 
highly suspicious tissue resides within the region. A warning can be provided if the 
virtual biopsy needle is outside the targeted position. A warning can also be provided 
if the proposed path of the biopsy needle will damage anatomical structures in the 
5 vicinity. In the case of a biopsy, the virtual biopsy needle can be modeled as a virtual 
cylinder. The tissue within the volume of the virtual cylinder can be removed from 
the region, examined in a separate display such as by texture analysis, and geometric 
features of the tissue, and then reinserted into the region, if desired. Virtual biopsy 
can be used in numerous applications, including the planning for examining 

1 0 pulmonary nodules, prostate tissue abnormality and breast cell calcification. 

Virtual biopsy can differentiate among tissue types using geometric 
feature analysis, texture analysis and/or tissue density analysis to determine the types 
of cells which are within the region of tissue extracted by the virtual biopsy needle. 

Yet another application of the present method is in the implantation of 

1 5 prosthesis or stent grafts. In this case, the virtual intervention includes the removal of 
damaged tissue in the targeted vicinity of an implant A virtual model of the implant 
can be optimized from an analysis of the targeted vicinity. The pathway for the 
insertion of the implant can be modeled as a generalized virtual cylinder. A warning 
can be provided if the generalized virtual cylinder placement will damage critical 

20 anatomical structures. The virtual model can be used to generate the actual insert. In 
addition, the resulting plan can be saved in a database for follow post-treatment 
evaluation of the actual intervention. Implants can take on various forms, such as 
cochlear implants, stent grafts and the like. 

The virtual cylinder is a generalized virtual volume which has the 

25 parameters of the tool it is simulating. In the case of the surgical planning for aural 

atresia, the tool is a drill bit and the virtual cylinder is a simple cylinder with a length, 
and circular cross section. However, the term virtual cylinder is not so limited. The 
diameter can vary as can the cross section and contour along the long axis. In 
addition, the virtual cylinder may be a deformable model, such as , for example, when 

30 the tool being modeled is a flexible catheter. 



Best Available Copy 



WO 01/56491 Page 7 of 41 



WO 01/56491 PCXAJS01/03746 

6 

. Yet another ap p lication-of.me_pr^entrnethod -ift-|ti-tfa^^ar\niTig-of 

carotid plaque analysis and intervention- In this case, the intervention includes the 
analysis of the plaque component on the arterial wall and the size and removal of 
plaque in a targeted vicinity. A virtual model of the plaque can be optimized from an 
5 analysis of the targeted vicinity. The pathway for the removal of the plaque can be 
modeled as a generalized virtual cylinder. A warning can be provided if the 
generalized virtual cylinder has a size or placement which will be ineffective at 
extracting the plaque if the plaque has a high risk of rupture or dislodgement in the 
present placement. The virtual model can be used to generate the arterial insert which 

10 will be used in the actual operation. In addition, the resulting plan can be saved in a 
computer database for future reference and post-treatment evaluation. 

A system for computer aided treatment planning includes a scanner, 
such as a CT or MRI scanner, for acquiring image data from a patient A processor is 
provided which receives the image data and generates a 3D, voxel based dataset 

1 5 representing at least a portion of the image data. A display, such as a liquid crystal 
display (LCD) or Cathode Ray Tube (CRT) is operatively coupled to the processor 
and provides a 3D representation the image data from the voxel based dataset A user 
interface device, such as a mouse or trackball, is operatively coupled to the processor 
and allows a user to manipulate the image on the display. Computer readable 

20 memory is operatively coupled to the processor and has a computer program stored 
therein. The computer program directs the processor to perform the steps of: 
applying a virtual intervention in the region of the 3D image in response to a signal 
from the user interface device; analyzing the intervention and automatically 
generating a warning indication if the intervention results in a high degree of risk 

25 arising. Previewing a contemplated treatment virtually, through the user interface, the 
user can modify the intervention to eliminate the warning. 

BRIEF DESCRIPTION OF THE DRAWING 

Further objects, features and advantages of the invention will become 
apparent from the following detailed description taken in conjunction with the 
30 accompanying figures showing illustrative embodiments of the invention, in which: 
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present method for computer aided treatment planning; 

Figure 2 is a simplified block diagram of a system suitable for 
performing the present method of computer aided treatment planning; 
5 Figure 3 is a simplified flow diagram illustrating an overview of 

exemplary rendering and virtual surgery steps used in the present method in the 
planning of treatment for aural atresia; 

Figure 4 is an exemplary subroutine for generating a binary space 
partitioning (BSP) tree data structure used for storing and manipulating image data in 
10 the present method; 

Figure 5 is a pictorial representation of a geometrical subdivision 
process of a node in the BSP-tree; 

Figure 6A is a schematic diagram of an exemplary image (illustrated in 
2D) partitioned by a number of leaf nodes; 
15 Figure 6B is a detail of one leaf node partition from Figure 6 A, further 

illustrating a region of overlap between a virtual cylinder placed in the image and the 
underlying tissue in the image; 

Figure 7 is a flow chart illustrating a voxel-based constructive solid 
geometry (CSG) subtraction operation which is employed in the virtual surgery 
20 operation of the present method; 

Figure 8 is a graphical representation of an exemplary screen display 
illustrating a 3D rendering of the structures of the ear with background tissue and the 
temporal bone not shown; 

Figure 9 is a graphical representation of an exemplary screen display 
25 illustrating a 3D rendering of the structures of the ear and illustrating the placement of 
a virtual cylinder as part of a surgical p lanning procedure for treating aural atresia; 

Figure 10 is a graphical representation of an exemplary screen display 
illustrating a 3D rendering of the structures of the ear and further illustrating the 
placement of a virtual cylinder as part of a surgical planning procedure for treating 
30 aural atresia; and 
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illustrating a 3D rendering looking down and through the auditory canal that is 
formed by performing a virtual drilling operation to remove the tissue occupied in the 
volume of the virtual cylinder. 
5 Throughout the figures, the same reference numerals and characters, 

unless otherwise stated, are used to denote like features, elements, components or 
portions of the illustrated embodiments. Moreover, while the subject invention will 
now be described in detail with reference to the figures, it is done so in connection 
with the illustrative embodiments. It is intended that changes and modifications can 
10 be made to the described embodiments without departing from the true scope and 
spirit of the subject invention as defined by the appended claims. 



DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

Figure 1 is a flow chart which illustrates an overview of the present 
method of computer aided treatment planning which is generally performed on a 

15 computer based system, such as that illustrated in Figure 2. The invention will be 
described in terms of medical applications performed on human patients and in the 
context of medical treatment, such as surgery, prosthesis implantation, biopsy, 
medication, therapeutic radiation, therapeutic ultrasound and the like. It will be " 
appreciated, however, that the invention is not limited to human patients, nor to the 

20 exemplary list of treatments referenced. The term treatment is used to mean an 

intervention in a region, such as but not limited to tissue, that is intended to effect an 
alteration of the region. 

Referring to Figure 1, the method includes the initial step of generating 
a three dimensional (3D) image representation of a region for which some form of 

25 medical treatment or intervention is contemplated (step 1 02). Generating such a 3D 
image representation generally involves acquiring a sequential series of 2D images, 
such as from a spiral computed tomography (CT) scanner or magnetic resonance 
imaging (MRI) scanner and trarisfonning this 2D image data into a volumetric data 
set which provides a 3D representation of the region on a 2D display, such as a 

30 computer monitor. Such a technique is well known in the art, and is discussed, for 
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example inU.S JRatent-No.-5,£H-^7^^ - 
incorporated by reference in its entirety. 

After the 3D image is presented to a user, such as a physician, some 
form of virtual intervention, which simulates at least a portion of a proposed 
5 treatment, is applied to the 3D image (step 104). The virtual intervention can take on 
several forms, such as the removal of tissue or artery plaques, the repair or 
reconstruction of a diseased or malformed organ, the placement of a prosthetic 
implant, the placement of a stent graft, the placement of biopsy needle, the 
placement of therapeutic radiation and the like. 
1 0 Using the resulting 3D image, and possibly the assistance of computer 

generated models of the applied intervention, the results of the virtual intervention can 
be evaluated and warnings can be generated indicative of high levels of risk attendant 
with the proposed intervention (step 106). Based on the displayed results, and any 
warnings provided, the user can repeatedly modify the proposed intervention (step 
15 108) and apply the modified intervention to the 3D image (step 1 04) until a 

satisfactory result is ascertained or it is determined that the proposed treatment is not 
feasible. Several alternative interventions can be saved in a database to compare the 
risks and efficacy of proposed alternative intervention plans. 

After the proposed intervention is finalized, the final intervention can 
20 be simulated and the results fully applied to the 3D image (step 1 10). The user can 
then view the results and navigate in and around the region to determine the efficacy 
of the proposed treatment (step 1 1 2). The planned results can then be used as a guide 
for the actual treatment with coordinate registration between the virtual model and the 
patient and as a gold standard to evaluate the actual intervention during post- 
25 intervention follow up examinations. 

Figure 2 is a simplified diagram of an exemplary system for 
performing the present computer aided treatment planning methods. In this 
exemplary embodiment, a patient 201 lies down on a platform 202 while scanning 
device 205 scans the area that contains the organ or organs which are to be examined. 
30 The scanning device 205 contains a scanning portion 203 which acquires image data 
of the patient and an electronics portion 206. Electronics portion 206 generally . 
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storing the scanning data, and a second interface 213 for sending data to the virtual 
navigation platform. Interface 207 and 213 can be included in a single interface 
component or can even be the same component The various operational components 
5 and subsystems in electronics portion 206 are connected together with conventional 
connectors. 

The data from the scanning portion 203 is generally in the form of a 
stack of two dimensional image slices of a region of interest, which are provided from 
conventional spiral computed tomography (CT) and magnetic resonance imaging 

10 (MRI) scanners. Central processing unit 209 converts the scanned 2D data to a 3D 
voxel data representation, in a manner known in the art, and stores the results in 
another portion of memory 211. Alternatively, the converted data can also be directly 
sent to interface unit 213 to be transferred to the virtual navigation terrninal 216. The 
conversion of the 2D data could also take place at the virtual navigation terminal 216 

1 5 after being transmitted from interface 213. Preferably, the converted data is 

transmitted over carrier 2 1 4 to the virtual navigation terminal 2 1 6 in order for an 
operator to perform the computer aided treatment pl ann ing. The data can also be 
transported in other conventional ways such as storing the data on a storage medium 
and physically transporting it to terminal 21 6 or by using satellite transmissions. " 

20 The scanned data need not be converted to its 3D representation until 

the visualization rendering engine requires it to be in 3D form. This may save 
computational steps and memory storage space. 

Virtual navigation terrninal 216 includes a screen 217 for viewing the 
image data, an electronics portion 215 and interface device 219 such as a keyboard, 

25 mouse or track ball. The electronics portion 215 generally includes a interface port 
221, a central processing unit 223, other components 227 necessary to run the 
terminal and a memory 225. The components in terminal 216 are connected together 
with conventional connectors. The converted voxel data is received in interface port 
221 and stored in memory 225. The central processor unit 223 then assembles the 3D 

30 voxels into a virtual representation which can be displayed on screen 217. Preferably, 
a graphics accelerator which is optimized for volume rendering can also be used in 
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using a high speed graphics work station, such as manufactured by Silicon Graphics, 
Inc., or in a high speed personal computer, such as an IBM compatible computer with 
a Pentium m (or higher) processor having a 1GHZ or faster clock speed. 
5 Hie operator can use interface device 219 to interact with the system 

200, such as to indicate which portion of the scanned body is desired to be explored. 
The interface device 219 can further be used to control the image being displayed, 
including the angle, size, rotation, navigational position and the like. 

Scanning device 205 and terminal 216, or parts thereof, can be part of 

10 the same unit. Numerous CT and MRI systems are suitable for such applications. A 
single platform may be used to receive the scanned image data, convert the image data 
to 3D voxels if necessary and perform the guided navigation. 

The method of Figure 1, while applicable generally to numerous 
treatment planning operations, will now be described in further detail in the context of 

15 surgical planning for the treatment of aural atresia. The surgical repair of aural atresia 
consists of several steps: drilling a new external auditory canal in the temporal bone; 
creating a new meatus in the concha; forming a new eardrum using temporalis fascia; 
and re-lining the bony canal with a skin graft. An important aspect of surgical 
planning in this regard is to define an acceptable path for the new auditory canal from 

20 the outer cortex to the inner ear and to visualize the important internal structures of 
the ear, including the relative spatial relationships among these structures. 

The present method of medical treatment planning provides for 3D 
imaging of the structures of the ear and allows interactive positioning of a virtual 
drilling site which will form the external auditory canal in the temporal bone. Using 

25 computer aided planning, this canal can be placed, analyzed and repositioned if 
necessary. If a number of proposed plans are acceptable, several variants can be 
saved in a database for comparison. Once the final location is accepted, then a virtual 
drilling operation can be performed to determine if the planned surgery is likely to be 
successful. Once the canal is virtually drilled, the user can navigate into and through 

30 the canal and visualize the surrounding structures of the ear. The operation is 

interactive and repeatable. The plan can be stored in the database and the surgeon can 
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and choose the best available plan. 

Figure 3 is a flow chart illustrating an overview of the present method 
for performing surgical planning in the context of treating aural atresia. In this case, 
5 the primary intervention thai the user, generally a surgeon, will undertake is the 

creation of a new external auditory canal which must be drilled through the temporal 
bone. Initially, a 3D rendering of the important structures of the ear is created using 
acquired 2D image scan data, such as from an MRI scan or CT scan (step 305). 
Because the various structures of the ear present a similar image intensity in the image 

10 scan data, it is preferable that the 3D rendering step include an image segmentation 
process to distinguish the various tissue types of the ear, such as the facial nerve, 
temporal bone and the like. From the segmented image data, a volume based 
rendering can be presented to the user, such as on a computer display, where the 
various structures of the ear are delineated, as is illustrated in Figure 8. 

15 Figure 8 is an exemplary system display which could be observed by 

the user on terminal 217. The display preferably takes the form of a graphical user 
interface (GUI) which allows the image to be fully manipulated on the display by the 
user operating the user interface device 219. The top of the display includes a tool 
bar, where commonly used commands can be readily referenced, in a manner known 

20 in the art. The right hand portion of the display shows a user interface portion 804, 

where image parameters can be observed and manipulated. The main display portion 
805 illustrates the segmented image data showing the various structures of the ear 
including the cochlea 806, stapes 808, malleus 810, and the facial nerve 812. The 
user can manually edit the segmented small structures, if the user desires to improve 

25 the segmentation results. By altering the transfer function used in the volume 

rendering process, the user can control the color and opacity of each of the structures. 
The user can also control non-geomtetry properties of each structure, such as shading, 
color and opacity. For example, as in the view shown, the user can render the 
temporal bone transparant to view the underlying anatomy of the ear. Alternatively, 

30 the temporal bone could be rendered transparent, such that the underlying anatomic 
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spatial relationship among these structures could be observed. 

As shown in Figure 9, a 3D cylinder is used as the model for the drill 
bit which a surgeon will use to form the external auditory canal. This cylinder 902 is 
5 interactively placed by the user in a position which is expected to open a passage 

through the temporal bone 904 from the outer bony cortex toward the middle ear (step 
3 1 0). The user can manipulate the length, diameter and position of the cylinder using 
the interface device 219 to drag the cylinder or by changing the parameters 
numerically in the cylinder control area 906 of the interface portion 804 of the 

1 0 display. The computer system then illustrates the placement of the cylinder, such as 
by displaying this region as a different color. By altering the opacity of the objects 
through which the cylinder 902 projects, the proximity of the cylinder to the structures 
of the inner ear can be clearly observed, such as is illustrated in Figure 10. 

The computer aided treatment terminal 216 can analyze the proximity 

1 5 of the cylinder in this initial placement to the critical structures surrounding the 
proposed drilling site, such as the facial nerve and ossicles, and issue warnings if 
drilling would result in potential damage to these structures (step 3 1 5). Based on the 
visual representation and the warnings that are presented, the user can alter the 
proposed intervention, such as by repositioning the cylinder, until a satisfactory path 

20 for the cylinder is determined or it is determined that the proposed treatment is not 

feasible. The user can also control the length and diameter of the cylinder to conform 
to different drilling depths and drill diameters, respectively. Once such a path has 
been determined, a virtual drilling operation can be performed by removing the region 
represented by the cylinder from the image (step 320). Of course, there may be more 

25 than one acceptable course of intervention. Accordingly, variations in the plan can be 
implemented and saved in a database such that alternatives can be compared and the 
best acceptable alternative may be selected. 

During and after the virtual drilling operation, the user can navigate 
through the bore of the drill site to visualize, from within the treated area, the efficacy 

30 and/or risks of the treatment (step 325). As illustrated in Figure 1 1 , when the virtual 
drilling operation is complete, the user can navigate completely through the new 
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auditory canal 1 1 02 and, at the evit point, visualize the spadal relationship-of-the 
newly formed canal with respect to the inner ear structures, such as the ossicular mass 
1 104. During any visualization step, the colors of the objects being viewed, as well as 
the relative opacity of the objects, can be varied by the user to present different views 
of the region. 

The virtual cylinder 902 is illustrated in Figures 9 and 10 as a simple 
cylinder having a circular cross section and a long axis. This simple model is a 
reasonable representation of the rigid drill bit used in the aural atresia surgical 
procedure. However, it will be appreciated, that the virtual cylinder need not be a 
simple cylindrical model and that it can and should be sized and shaped to conform to 
the size and shape of the iristrument that will be used to perform the contemplated 
treatment. The virtual cylinder need not have a circular cross section nor a constant 
diameter. For example, if the tool is generally shaped as a curved rod, a cylindrical 
model which is curved along its major axis would be appropriate. For a complex tool 
with articulated joints and extensible members, a more complex 'Virtual cylinder" 
model can be used. In addition, the generalized virtual cylinder can be modeled as a 
deformable object to fit in a specific anatomical environment and to model flexible 
tools, such as catheters. Regardless of the actual model employed, the principle of the 
method remains the same. 

A difficulty encountered in the imaging step 305 is that several of the 
relevant anatomical structures have similar intensity values on the CT image. This, in 
combination with the complex anatomy of the ear, can make it difficult to distinguish 
the various structures. To address this problem, a tworlevel image segmentation 
process can be employed. The two-level segmentation process involves low-level 
processing of the voxels in the region of interest followed by high-level organ 
extraction. During the low-level processing, the voxels of the 3D dataset are clustered 
into groups based on an intensity feature of the voxels, which can be measured by an 
associated local intensity value vector. This can be determined using a modified self- 
adaptive on-line vector quantization algorithm, such as is described in the article CK A 
self-adaptive on-line vector quantization algorithm for MRI segmentation," by Chen 
et al. in the proceedings of The 7th Scientific Meeting of ISMRM, May 1999, 
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classification, each voxel is associated with a local vector which is defined in 3D 
space. From the local vectors, a feature vector series can be derived using a 
components analysis which is well known in the art The feature vectors are then 
5 clustered using a self-adaptive on-line vector quantization algorithm. The voxels are 
then grouped according to the classification of their feature vectors and are assigned 
an integer value representing this classification. 

After the low-level processing is complete, the high level organ 
extraction processing can follow. Initially, a user locates a seed, or starting point, 
10 within regions representative of soft tissue, bone and air spaces. The system then 
applies a region growing algorithm starting from the seed points to extract the 
anatomical features of the ear, such as the temporal bone, stapes, ossicles and facial 
nerve. 

Hie temporal bone, which presents high contrast compared to the 

15 surrounding tissue is fairly easy to automatically segment. However, certain 

structures, such as the inner ear and facial nerve, may require addition user input to 
fully delineate these structures. For example, the soft tissue of the inner ear presents a 
similar intensity value on CT images as compared to the surrounding soft tissue. 
Thus, to insure proper extraction of this feature, it may be desirable for the user to 

20 manually delineate the outline of this structure by manually tracing the contour on one 
or more of the image slices. Similarly, extraction of the facial nerve may also require 
manual intervention from the user, such as by manually identifying the outline of the 
facial nerve in each image slice in which this structure is present. 

While the above described two level image segmentation is preferred, 

25 any method which provides accurate delineation of the neighboring structures in a 

region of interest can be used in the practice of the present treatment planning method. 
One such technique is described in the article "On segmentation of colon lumen for 
virtual colonoscopy" by Liang et al., Proceedings of SPEE Medical Imaging, pp 270- 
278, Feb. 1999, San Diego. 

30 Once image segmentation is performed, 3D image generation can be 

performed for each of the segmented objects using a number of known techniques, 
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surface. However, because of the complexity of the structures which make up the ear, , 
interactive rendering of all polygons in the display for each change to a portion of the 
display is processor intense and unduly costly. As more colors and surfaces are 
5 delineated in the displayed image, this burden increases. To minimize the processing 
overhead, the volume image dataset can be stored in a partitioned data structure, such 
as a binary space-partitioning (BSP) tree, in which the large dataset is parsed into 
relatively small portions which are stored in leaf nodes of the data structure. By 
identifying which leaf nodes are effected by any given operation, and only performing 

10 operations, such as Constructive Solid Geometry (CSG) operations, on the effected 
leaf nodes, the processing burden for interactive operations can be significantly 
reduced. As will be set forth in more detail below, the processing burden can be 
further reduced by use of a level of detail (LOD) rendering mode and/or a wavelet . 
transformation to reduce the data volume. 

1 5 An exemplary BSP-tree construction will be described in connection 

with Figures 4 and 5. The BSP-tree uses the original volume image data as the root 
node of the tree. This is represented by (0,0,0, dim x, dim y, dim z, n p ) where the 
parameters dim x, dim y 9 and dim z are the resolution of the original dataset and n,, is 
the total number of polygons within the original dataset. A binary subdivision is " 

20 performed on this node that geometrically divides the node into two sub-nodes if n p is 
greater than a predetermined constant n^. The binary subdivision operation is 
recursively applied to each sub-node until the number of polygons in the sub-nodes is 
less than or equal to the value of no- At this point, the resulting sub-nodes are defined 
as leaves in the constructed tree data structure. 

25 The node subdivision which is used in the tree generation process is * 

illustrated in the algorithm illustrated in Figure 4 and the pictorial diagram of Figure 
5. Referring to Figure 5, (xO, yO, zO, xl, yl, zl, np) represents a node in the tree in 
Cartesian space. The subdivision process occurs by dividing the node using a 
Cartesian plane which is perpendicular to the longest axis of the given node. In the 

30 case of Figure 5, the longest axis of the node shown is along the X axis. The position 
at which the node will be bifurcated is detennined by accumulating a number of 
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accumulated number, n^, reaches a value of rij/2. This exemplary construction of the 
BSP tree results in a substantially balanced tree structure, both in tree depth and the 
number of polygons in the resulting leaf nodes. 
5 Figure 6A illustrates the placement of the virtual cylinder 602 of Step 

310, represented as a voxel based volume, into a portion of the volume image 600 
including a portion of the temporal bone 601. For the sake of simplicity, the drawing 
in Figure 6 A and 6B are illustrated in two dimensions, however, it will be appreciated 
that the image is a 3D voxel representation. The image 600 is illustrated as 

10 partitioned by the leaf nodes 604, 606, 608, 610 and 612 of the BSP data structure. 
To determine the effect of the placement of cylinder 602, the leaf nodes which are 
effected by the placement of the cylinder 602 are identified, then a voxel-based 
constructive solid geometry (CSG) subtraction operation is performed. 

Referring to Figure 7, to identify the leaf nodes which are effected by 

15 the placement of the cylinder 602, a minimum, axis aligned bounding box 614 is 

generated which encompasses the cylinder (step 702). Next, the bounding box 614 is 
partitioned into blocks 616, 618, 620, 622, 624 according to the boundaries of the leaf 
nodes (step 704). A first block is selected (step 706) and the intersection 624 of the 
cylinder 602 and temporal bone 601 within the selected block 604 is determined (step 

20 708). It is within this intersection region 624 that a CSG subtraction operation and 
voxel regeneration operation will take place. Because the voxels in the intersection 
region 624 within the block will be altered by a CSG subtraction operation, the 
original voxels are stored as a backup so that the last image can be restored, for 
example, if the cylinder 602 is subsequently repositioned (step 710). Once the 

25 original values are stored in a back-up memory location, a subtraction of the voxel 
values of the cylinder 602 from the voxel values of the temporal bone 601 in the 
intersection region 624 detennines the shape after a drilling operation (step 7 12). A 
regeneration process in the area of intersection region provides the new surface 
description (step 714). From the new surface description, the new surface can be 

30 rendered in the 3D image of the block, such as by using a Marching Cubes algorithm 
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-(stepJ7J6).-The4>rocess4rom-st^ within 
the bounding box 614 (step 718). 

To further manage the data in a manner which allows efficient navigation and 
viewing of the virtual organs being displayed, a level-of-detail (LOD) rendering mode 
can be used. In the LOD method, a reduced dataset is generated from the full volume 
data set. For example, the 5 12x5 12x256 full dataset can be reduced to a 64 x 64 x 32 
reduced volume dataset using a multi-resolution decomposition with three levels of 
thresholding. Next, polygons used to render the volume images in both the enlarged 
and reduced volume datasets can be extracted. A traditional Marching Cubes method 
can be used to extract polygons to fit the surface of the object. 

During navigation or viewing, polygon culling can be applied by first 
removing those leaf nodes that are completely outside the field-of-view from current 
processing operations. The remaining polygons are recalled from the BSP tree, 
ordered and rendered in those spaces which were not culled. Thus, the BSP tree 
provides an effective tool for selecting a relevant portion of the dataset for a particular 
navigation or display operation. 

The enlarged and reduced datasets are cooperatively used in a two 
level LOD rendering mode. If a user is interacting with the object being displayed, 
such as rotating, shifting or effecting other changes in the field of view, the polygons 
from the reduced dataset (64-sized) are rendered. Because of the significantly lower 
number of polygons involved, interaction with the reduced dataset volume can be 
performed faster and with less processing overhead. The tradeoff for the increased 
speed is reduced image resolution. If there is no interaction from the user after a 
predetermined time period, the polygons of the enlarged dataset (5 12 -sized) are 
selected from the BSP tree and are rendered to provide a high resolution image of the 
current field of view 



or at least desirable, to reduce the size of the dataset to speed up processing and 
reduce processing cost Noting that the tree structure can be preserved within a range 
of scales, the large volume can be shrunk to a smaller scale space for structure 
analysis. 




As noted above, when a large dataset is involved, it may be required, 
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used. The input data is the stack of binary images of the same size which can be 
obtained from the segmentation results of the CT or MRI scan. The x-direction is 
taken along the slice image width, the y-direction is along the slice image height, and 
5 the z-direction is along the direction of slice by slice. The foreground voxels in the 
tree volume are set to value of 128 (maximum) and the background voxels are set to 
value 0 (minimum). A Daubechies' bi-orthogonal wavelet transform with all rational 
coefficients can be employed. This one-dimensional (ID) discrete wavelet 
transformation (DWT) is first applied along to the x-direction row by row. From 

10 application of the DWT only the lower frequency components are retained and 

packed. The computation is preferably implemented in floating points. Noting that 
the DWT is applied to the binary signal, there are two kinds of nonzero coefficients 
which result in the lower frequency component. The first is of value 128 and this 
kind of coefficients are located in the interior of the volume. The second is of a value 

15 not equal to 128 and these coefficients locate the boundary of the volume. 

The coefficients of the second kind are compared against a 
predetermined threshold value. If its absolute value is larger than a pre-set threshold 
Tl, the value of the coefficient is set to 128; otherwise, it is set to 0. This results in a 
stack of binary images with a row size of half of the original dataset. The same DWT 

20 is then applied to the resulting dataset along the y-direction column by column, where 
the similar thresholding is employed to the lower frequency components. The result 
is again a stack of binary images, but now with both half row and column size as 
compared to the original dataset. Finally, the DWT is applied to the last result along 
the z-direction and the lower frequency components are retained. This step completes 

25 the first level decomposition. 

The resulting dataset of the first level decomposition is of half size in 
all three directions as compared to the original dataset. If the sminking procedure 
stops at this level, the finial thresholding is applied. It revalues those coefficients of 
nonzero and non-128 value. If the absolute value of this kind of coefficient is larger 

30 than a pre-set threshold T2, it will be revalued as 128; otherwise, it is revalued as 0. 
If further shrinking is needed, the same thresholding algorithm is applied with the 
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the dataset shrunk at the last previous level. The decomposition procedure can be 
recursively applied until the resulting volume meets the desired reduced data volume. 
In the case where the slice images are of 5 12X5 12 pixel size, the maximum 
5 decomposition level is usually three, resulting in a 64x64 reduced pixel size. 

The volume is i so tropically shrank in all directions with the presented 
method. The two pre-set thresholds, Tl and T2, are used to control the degree of 
shrinking. If the volume is significantly over shrunk, connectivity may be lost in the 
reduced volume. If it is over shrunk to a leaser degree, two separate branches may 

10 merge into one branch in the reduced volume dataset The larger the two threshold 

values, the thinner the reduced volume is. The range of those two thresholds is [0, r x 
128], where 0<r<l . Preferably, the range for virtual endoscopy is re (0.08, 0.28) for 
Tl and re (0.7, 0.98) for T2. The exact determination is dependant on the feature size 
of the particular application and is selected to achieve reduction while retaining the 

15 fidelity of the structure information in the shrunk volume.. 

To determine if a warning should be generated in step 315 (Figure 3), 
each block 61 6, 61 8, 620, 622, 624 can be evaluated to determine if tissue types, other 
them that of the temporal bone, are proximate the current location of the cylinder 602. 
If the cylinder is within a predetermined threshold distance of a critical structure, a 

20 warning can be generated. For example, if the proposed cylinder location is within 1- 
2 mm of the facial nerve, a warning can be provided. The warning can take the form 
of a text message, an audible signal, or by highlighting the area of risk in the image 
for the user, such as by vise of a special color or by automatically magnifying the 
region for closer examination by the user. 

25 While the invention has been described in detail in connection with the 

treatment planning for aural atresia, it is generally applicable to numerous treatment 
planning operations both in the ear and elsewhere in the body. For example, in the 
case of angioplasty, the surgeon inserts a catheter into an occluded artery and inflates 
a balloon at the end of the catheter to force the occluded artery open and to expand a 

30 stent which maintains the opening. While this has become a common procedure, it is 
not without risk. For example, the arterial occlusion is generally related to a build up 
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of plaque and-fatty-deposits^n-the^ are 
dislodged during the angioplasty process, there is a risk of stroke and other 
complications. Using the present method of treatment planning, the artery can be 
imaged and, through image segmentation, the quantity and nature of the plaque 
deposits can be determined. The severity of the occlusion can be viewed by the 
surgeon who can navigate in the 3D image within the artery. A virtual intervention 
can then be performed, i.e., placing a virtual catheter within the arterial volume and 
expanding a virtual stent, and the results observed. If problems are observed, the user 
can then alter the course of treatment to mimmize the risk. Unlike the case of aural 
atresia, in this case, the virtual catheter would require a dynamic model that conforms 
to the contours of the interior surface of the arterial wall. Such a model is analogous 
to the force field model previously used in guiding a virtual camera along a fly path in 
performing virtual colonoscopy. 

Similarly, the present method is applicable to treatment planning for 
the formation and implantation of a stent graft for treating abdominal aortic 
aneurisms. The 3D imaging can be used to determine the size, location and nature of 
the aneurism, in addition, using image segmentation, the quality of the arterial wall 
can be determined by analyzing the composition of the arterial wall to deteimine the 
degree of plaque build up and stenosis. A virtual stent graft can be modeled to fit in 
the region of the AAA and the graft can be inserted into the 3D image. Alternatively, 
the surgical removal of plaque can be the modeled intervention. In either case, the 
user can navigate within the treated region to visualize the results of the proposed 
intervention. 

Another application of the present method of treatment planning is in a 
virtual biopsy. In this case, an organ, such as the prostate, breasts or lungs can be 
scanned and rendered as a segmented 3D image. Preferably, the image segmentation 
process at least partially highlights those portions of the organ of interest which have 
a high likelihood of containing cancerous tissue. A virtual cylinder can be placed into 
this region to simulate the insertion of a biopsy needle. The position, size and shape 
of the cylinder can be optimized by the user to insure that at least part of the 
suspicious region is within the volume of the cylinder. The region within the virtual 
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the display for further analysis. 

Known volume rendering techniques use one or more defined transfer 
functions to map different ranges of sample values of the original volume data to 
5 different colors, opacities and other displayable parameters for navigation and 
viewing. During navigation, the selected transfer function generally assigns 
maximum opacity to the wall of the object being viewed. However, once a suspicious 
area is detected during virtual exaniination and is extracted during the virtual biopsy 
intervention, the physician can interactively change the transfer function assigned 

10 during the volume rendering procedure such that the outer surface being viewed 

becomes substantially transparent, allowing the interior structure of the region to be 
viewed. Using a number of predetermined transfer functions, the suspicious area can 
be viewed at a number of different depths, with varying degrees of opacity assigned 
throughout the process. In addition, the shape of the region and texture of the region 

1 5 undergoing virtual biopsy can be analyzed to determine a likelihood of cancerous 
tissue in the region being biopsied. 

Although the present invention has been described in connection with 
specific exemplary embodiments, it should be understood that various changes, 
substitutions and alterations can be made to the disclosed embodiments without 

20 departing from the spirit and scope of the invention as set forth in the appended 
claims. 
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CLAIMS 

1 . A method of computer aided treatment planning comprising; 

generating a three dimensional image of a region including at least one 
anatomical structure for which treatment is contemplated; 
5 applying a virtual intervention in said region of said three dimensional 

image; 

deteimiriing the effect of said intervention; 
modifying the intervention for improved treatment results; and 
navigating within the three dimensional image in the area where the 
10 intervention is applied to visualize the results of the intervention. 

2. The method of computer aided treatment planning according to Claim 1, 
wherein the region includes the ear and the at least one anatomical structure includes 
at least one of the temporal bone, facial nerve, and stapes. 

3. The method of computer aided treatment planning according to Claim 2, 
15 wherein the virtual intervention includes the placement of a virtual cylinder 

representing the location where an external auditory canal may be formed. 

4. The method of computer aided treatment planning according to Claim 3, 
wherein the step of deterniining the effect of the virtual intervention includes 
determiriing the proximity of the virtual cylinder to anatomical structures of the ear 

20 and providing a warning if said proximity is less than a predetermined threshold 
distance. 

5. The method of computer aided treatment planning according to Claim 4, 
wherein the step of modifying the intervention includes changing the position of the 
virtual cylinder. 
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wherein the step of modifying the intervention includes changing the diameter of the 
virtual cylinder. 

7. The method of computer aided treatment planning according to Claim 4, 

5 wherein the step of modifying the intervention includes changing the length of the 
virtual cylinder. 

8. The method of computer aided treatment planning according to Claim 3, 
further comprising the steps of virtually removing the volume within the virtual 
cylinder and navigating through the region in the three dimensional image. 

10 9. The method of computer aided treatment planning according to Claim 1, 
further comprising the step of saving the results of a planned intervention in 
computer readable storage and comparing the results of an actual intervention to the 
saved results of the planned intervention. 

10. The method of computer aided treatment planning according to Claim 1, 
15 wherein the step of generating a three dimensional image of a region further 

comprises: 

acquiring a set of 2D images of a region; 
converting said 2D images to a voxel based dataset of region; 
storing the voxel based dataset in a partitioned dataset; 
20 rendering the three dimensional image from the voxel based dataset. 

11. The method of computer aided treatment planning according to Claim 10, 
further comprising the steps: 

performing automated segmentation of the voxel based dataset 
to identify features; and 
25 manually editing and extracting the automated segmented 

features, wherein the rendering operation employs both the voxel based dataset and 
said identified features. 
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12. The method of computer aided treatment planning according to Claim 11, 
wherein said rendering operation further comprises a level-of-detail operation and a 
constructive solid geometry operation. 



13. The method of computer aided treatment planning according to Claim 10, 

5 wherein the partitioned dataset is a binary space partitioning tree having a plurality of 
leaf nodes wherein a portion of the voxel based dataset are stored and wherein said 
step of applying the intervention further comprises: • 

identifying those leaf nodes which are effected by said intervention; 
applying said intervention to the effected leaf nodes; and 
10 re-rendering only the portions of the three dimensional image 

associated with the effected leaf nodes. 

14. A method of computer aided treatment planning comprising: 

generating a three dimensional image of a region including at least one 
15 anatomical structure for which treatment is contemplated; 

applying a virtual intervention in said region of said three dimensional 

image; 

analyzing the intervention and automatically generating a warning 
indication if said intervention results in a high degree of risk from arising; and 
20 modifying the intervention to eliminate said warning. 

15. The method of computer aided treatment planning according to claim 1 4, 
wherein the intervention is performed by a tool having a tool contour and wherein 
said step of applying the intervention includes applying a generalized virtual cylinder 
substantially conforming to the tool contour to the region. 

25 16. The method of computer aided treatment planning according to claim 15, 

wherein the tool has elastic properties and the generalized virtual cylinder is further 
modeled as a deformable object. 
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wherein the region includes the ear and the at least one anatomical structure includes 
at least one of the temporal bone, racial nerve, ossicles and stapes. 

1 8. The method of computer aided treatment planning according to Claim 17, 
5 wherein the virtual intervention includes the placement of a virtual cylinder 

representing the location where an external auditory canal may be formed. 

1 9. The method of computer aided treatment planning according to Claim 1 8, 
wherein the step of analyzing the effect of the virtual intervention includes 
deterniining the proximity of the virtual cylinder to anatomical structures of the ear 

10 and said warning is provided if said proximity is less than a predetermined threshold 
distance. 

20. The method of computer aided treatment planning according to Claim 19, 
wherein the step of modifying the intervention includes changing the position of the 
virtual cylinder. 

15 21. The method of computer aided treatment planning according to Claim 1 9, 

wherein the step of modifying the intervention includes changing the diameter of the 
virtual cylinder. 

22. The method of computer aided treatment planning according to Claim 1 9, 
wherein the step of modifying the intervention includes changing the length of the 
20 virtual cylinder. 



23. The method of computer aided treatment planning according to Claim 1 8, 
further comprising the steps of virtually removing the volume within the virtual 
cylinder and navigating through the region in the three dimensional image. 
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further comprising the step of saving the treatment plan in computer readable media 
and comparing the treatment plan to the results after intervention. 

25. The method of computer aided treatment planning according to Claim 14, 
5 wherein the intervention is the placement of a virtual biopsy needle into a targeted 

position within the region, and wherein said warning is provided if said virtual biopsy 
needle is outside the targeted position. 

26. The method of computer aided treatment planning according to Claim 14, 
wherein the intervention is the placement of a virtual biopsy needle into a targeted 

10 position within the region, and wherein said warning is provided if said virtual biopsy 
needle may damage any anatomical structure proximate the targeted position. 

27. The method of computer aided treatment planning according to Claim 26, 
wherein the targeted region includes at least one of a prostate, breast and a lung. 

28. The method of computer aided treatment planning according to Claim 16, 

15 wherein the intervention is the placement of an arterial stent and wherein the tool is a 
balloon catheter. 

29. The method of computer aided treatment planning according to Claim 28, 
wherein the step of generating a three dimensional image of a region includes an 
image segmentation step for differentiating the tissue of arterial wall from plaque 

20 build up on the arterial wall. 

30. The method of computer aided treatment planning according to Claim 29, 
further comprising the step of quantifying the tissue components of plaque build up 
on the arterial wall. 
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wherein the intervention is the placement of a prosthetic implant and the step of 
applying the virtual intervention includes: 

generating a virtual model of the prosthetic implant; 
5 applying a generalized virtual cylinder in the proposed location of the 

implant; and 

placing the virtual model of the prosthetic implant within the volume 
of the generalized virtual cylinder. 

32. The method of computer aided treatment planning according to Claim 3 1 , 
10 wherein the step of modifying the intervention includes altering at least one of the 

size, shape and position of the generalized virtual cylinder. 

33. The method of computer aided treatment planning according to Claim 32, 
wherein the warning is generated if the placement of the generalized virtual cylinder 
will impinge on nearby anatomic structures. 

15 34. The method of computer aided treatment planning according to Claim 33, 

wherein the step of modifying the intervention includes altering the virtual model of 
the prosthetic implant. 

35. The method of computer aided treatment planning according to Claim 34, 
wherein the implant is a stent graft, and wherein the virtual intervention includes 
20 placing a virtual model of a stent graft within a virtual model of an aortic blood 
lumen. 



36. The method of computer aided treatment planning according to claim 35, 
wherein the step of modifying the intervention includes altering the virtual model of 
the stent graft. 
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further comprising the step of providing the virtual model to a computer assisted 
system for fabricating the prosthetic implant 

38. The method of computer aided treatment planning according to claim 16, 

5 wherein the intervention is the removal of plaque from an arterial surface and the step 
of applying the virtual intervention further comprises: 

generating a virtual model of the plaque with size and components 
within the arterial volume; 

applying a generalized virtual cylinder in the proposed location of the 

10 artery; and 

placing the virtual model of the plaque within the volume of the 
generalized virtual cylinder. 

39. The method of computer aided treatment planning according to claim 38, 
wherein the step of modifying the intervention includes altering at least one of the 

15 size, shape and position of the generalized virtual cylinder. 

40. The method of computer aided treatment planning according to claim 38, 
wherein the warning is generated if the placement of the generalized virtual cylinder 
will impinge on nearby anatomic structures. 

41 . The method of computer aided treatment planning according to claim 38 

20 wherein the step of modifying the intervention includes altering the virtual model of 
the plaque. 

42. A system for computer aided treatment planning comprising: 
a scanner for acquiring image data from a patient; 

a processor, said processor receiving the image data and generating a voxel 
25 based dataset representing at least a portion of the image da ta; 



Best Available Copy 



WQ 01/56491 . Page 31 of 41 



WO 01/56491 PCT/USO 1/03746 

30 

a-display-df>erativdy-covipled-to-said processor-and-providing-a-three 

dimensional representation of the at least a portion of the image data from said voxel 
based dataset; 

a user interface device operatively coupled to the processor, the three 
5 dimensional representation being manipulated in response to input by a user via said 
user interface device; and 

a* memory operatively coupled to said processor and having a computer 
program stored therein, said computer program directing the processor to perform the 
steps: 

10 applying a virtual intervention in said region of said three 

dimensional image in response to a signal from said user interface device; 

analyzing the intervention and automatically generating a 
warning indication if said intervention results in a high degree of risk from arising;, 
and 

15 modifying the intervention to eliminate said warning. 
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